Towards Reliable Convergence in the Training of Neural Networks - The Streamlined Glide Algorithm and the LM Glide Algorithm

Multilayer perceptron neural networks continue to be very useful in many fields. Unfortunately training these networks is still a major problem due to lack of convergence. This converge problem stems from the fact that the error curve has vast flat regions, making even higher-order algorithms move the network weights very slowly. In an ongoing series of papers including this one we propose a new class of training algorithms which is a promising solution to this non-convergence problem. These algorithms, called the Glide Algorithms, are based on the simple idea of changing the weights quickly in flat regions. This paper presents a streamlined version of the basic Glide Algorithm and a new version that incorporates Levenberg-Marquardt type moves. According to experimental results obtained so far both algorithms have a 100% convergence rate with relatively low standard deviation in the convergence time.